Image Processing Device and Method

ABSTRACT

The present invention provides image processing device and method for effectively generating a difference image from plural images. The plural images generated by an image generation unit are first associated with radiography date and time information and stored in a storage unit, at least one reference image and one comparison image are designated by an image designation unit from the stored images, the date and time information of the designated reference image is compared with the date and time information of the designated comparison image by an image comparison unit, a difference process is executed by a difference processing unit based on an operation determined based on the comparison result, and the processed difference image is displayed on a display unit under the control of a display control unit.

TECHNICAL FIELD

The present invention relates to image processing device and method,more particularly, to image processing device and method which processplural medical images.

BACKGROUND ART

In recent years, use of digital images is advanced in the field ofdiagnostic imaging or medical imaging. For example, a device whichradiographs or takes an X-ray image by using a semiconductor sensor hasthe advantages as compared with a radiography system which usesconventional silver salt photography. More specifically, in such adigital radiography device, it is possible to record the image extendingover an extremely wider radiation exposure region, and it is alsopossible to more easily construct an efficient system with respect toimage storing and transfer.

Moreover, by digitizing a medical image, a possibility of diagnosticform which was difficult in conventional diagnosis using silver saltphotograph comes out. That is, in the conventional diagnosis, in a casewhere plural X-ray photographs which were radiographed at differentpoints in time during observation of patient's condition are comparedfor diagnosis, the films on which the X-ray photographs have beenrespectively developed are generally hung on a light box (schaukasten),and the hung films are actually compared and read.

Meanwhile, in the case where the digital images are used in thediagnosis, the two digital images which were radiographed at differentpoints in time with respect to one patient are subjected to registrationso that the normal anatomical structure on one digital image conforms tothat of the other digital image, and then a difference process isexecuted to the two digital images, whereby a difference image isgenerated and output. Subsequently, the output difference image iscompared with the pair of the two original digital images, whereby it ispossible to more accurately grasp a change between the two originalimages.

FIG. 11 is a block diagram showing the constitution of a conventionaldifference image generation and display device. In FIG. 11, a referenceimage and a comparison image which are input to a density correctionunit 401 are both equivalent to medical image data of a specific regionwhich was radiographed at different points in time. In the densitycorrection unit 401, the image data corresponding to these images arecorrected so that the distributions of the density values of the imagedata of these images become substantially the same. In a registrationunit 402, the local relation of the anatomical structures of thespecific region are acquired. Then, in a deformation unit 403, eachpixel on one of these images, that is, the comparison image here, isdeformed so as to overlap the corresponding pixel on the referenceimage, and, in a difference operation unit 404, a difference process isexecuted between the corresponding pixels to generate a differenceimage. Subsequently, the generated difference image is displayedtogether with the reference image and the comparison image on an imagedisplay unit 7. For example, such an operation is disclosed in JapanesePatent Application Laid-Open No. 10-155746 which corresponds to U.S.Pat. No. 5,987,345.

Here, the order of operation for the reference image and the comparisonimage in the above difference process is previously set. Morespecifically, if it is assumed that one of these images is a past imageand the other thereof is a current image, the order of operation forthese images is set to “past image”→“current image”, or “currentimage”→“past image”.

Typically, the difference image is used to extract a change whichappears in the subject on the images which were radiographed atpredetermined intervals in, for example, routine medical examinations.In such a use, as described above, a pair of images is used as thetarget, and it only has to execute the difference process to each ofthese images in the predetermined order.

However, in case of observing the progress of a specific patient, it isnecessary to radiograph the target plural times during a relativelyshort period of time, sequentially observe the results of somethingmedical treatment, and further grasp the progress of the relevantmedical treatment by tracing the past career. In such conventionalprogress observation, there is no method of effectively generating adiachronic difference image.

DISCLOSURE OF THE INVENTION

In consideration of the above conventional situation, the presentinvention has an object to provide an image processing device which canexecute an effective difference process in progress observation, animage processing method which is applied to the above image processingdevice, a program which is used to cause a computer to execute the aboveimage processing method, and a storage medium which can store thereinthe above program.

To achieve the above object, the image processing device according tothe present invention is characterized by comprising: a storage unitadapted to store images; an image designation unit adapted to designatea reference image and a comparison image from among the images stored bythe storage unit; a comparison unit adapted to acquire radiography dateand hour information of the designated reference image and thedesignated comparison image, and compare the radiography date and hourinformation of the reference image with the radiography date and hourinformation of the comparison image; a determination unit adapted todetermine an operation in case of generating a temporal difference imagebased on a comparison result by the comparison unit; and a differenceimage generation unit adapted to generate the temporal difference imagefrom the reference image and the comparison image by using the operationdetermined by the determination unit.

Moreover, the image processing method according to the present inventionis characterized by comprising: an image designation step of designatinga reference image and a comparison image from among plural images storedin a storage unit; a comparison step of acquiring radiography date andhour information of the designated reference image and the designatedcomparison image, and comparing the radiography date and hourinformation of the reference image with the radiography date and hourinformation of the comparison image; a determination step of determiningan operation in case of generating a temporal difference image based ona comparison result in the comparison step; and a difference imagegeneration step of generating the temporal difference image from thereference image and the comparison image by using the operationdetermined in the determination step.

An image processing device according to the present invention forachieving the above object is characterized by an

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the whole constitution of an imageprocessing device according to the present invention;

FIG. 2 is a block diagram showing the constitution of a system to whichthe image processing device according to the present invention isapplied;

FIG. 3 is a basic flow chart of an image processing method according tothe present invention;

FIGS. 4A and 4B are schematic diagrams showing an example of an imageselection screen according to the first embodiment of the presentinvention;

FIGS. 5A and 5B are schematic diagrams showing an example of the imageselection screen according to the first embodiment of the presentinvention;

FIGS. 6A and 6B are diagrams for explaining the combinations of imagesfor generating a difference image;

FIGS. 7A and 7B are diagrams for explaining the relation between achange of shadow and a difference signal;

FIGS. 8A and 8B are schematic diagrams showing an example of shadowextraction on the difference image;

FIG. 9 is a schematic diagram showing an example of an image selectionscreen according to the second embodiment of the present invention;

FIGS. 10A and 10B are schematic diagrams showing an example of an imageselection screen according to the third embodiment of the presentinvention;

FIG. 11 is a block diagram showing the basic constitution of aconventional difference process;

FIGS. 12A and 12B are diagrams for explaining registration of images,according to the present invention;

FIGS. 13A and 13B are schematic diagrams showing an example of the imageselection screen according to the first embodiment of the presentinvention;

FIGS. 14A and 14B are diagrams for explaining the combinations of imagesfor generating the difference image; and

FIGS. 15A and 15B are diagrams for explaining the relation between achange of shadow and the difference signal.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the embodiments to which the present invention is appliedwill be explained in detail with reference to the attached drawings.

First Embodiment Constitution of Image Processing Device

FIG. 1 is a block diagram showing the functional constitution of theimage processing device according to the first embodiment of the presentinvention. Initially, the whole operation of the image processing devicewill be schematically explained with reference to FIG. 1.

The image processing device according to the present embodiment consistsof an image generation unit 1 for generating an image, an image storageunit 2 for accumulating and storing the generated images, an imagedesignation unit 5 for designating at least one reference image and onecomparison image from the images accumulated and stored in the imagestorage unit 2, an image comparison unit 3 for acquiring the respectiveradiography times of the designated reference and comparison images andspecifying the context of the acquired radiography times, a differenceprocessing unit 4 for generating a temporal difference image from thereference and comparison images based on the specified result by theimage comparison unit 3, a display control unit 6 for controlling adisplay of the difference image, and an image display unit 7 fordisplaying the reference, comparison and difference images under thecontrol of the display control unit 6.

In FIG. 1, the medical images generated by the image generation unit 1are accumulated and stored in the image storage unit 2. Then, a pair orplural pairs of the stored images are read according to need, thedifference process is executed to the read images by the differenceprocessing unit 4, the difference image acquired as the result of thedifference process is displayed on the image display unit 7 through thedisplay control unit 6, and the displayed image is used in diagnosis.

Meanwhile, the former images used in the generation of the differenceimage are similarly displayed independently or together with thedifference image on the image display unit 7 under the control of thedisplay control unit 6, and the displayed images are then used indiagnosis.

In the above operation, the image comparison unit 3 compares the imagesbeing the target of the difference process by the later-describedmethod, thereby controlling the process of the difference processingunit 4. Meanwhile, the image designation unit 5 designates the imagesbeing the target of the difference process by the later-describedmethod, thereby determining the images to be output from the imagestorage unit 2 to the difference processing unit 4 based on thedesignation.

For example, by achieving such functions, as shown in FIG. 2, an imagegeneration device 200 and a file server 300 which are connected to acomputer (PC) 100 through a network 400 can be used. In the drawing,various peripheral devices are connected through a bus 107 to a CPU(central processing unit) 101 which is provided in the computer 100.Moreover, the computer 100 can transmit/receive image data to/from theimage generation device 200 and the file server 300 both externallyconnected through the network 400 through a not-shown interface.Incidentally, a magnetooptical disk 110, a mouse 120, a keyboard 130, aprinter 140 and a display device 150 are connected as the peripheraldevices to the computer 100, and an accelerator 105, a hard disk 104, aRAM 102 and a ROM 103 are provided as well as the CPU 101 in thecomputer 100.

In the above constitution, the image generation unit 1 of FIG. 1 can beassociated with the image generation device 200 of FIG. 2 whichgenerates and outputs a digital image signal indicating a subject (notshown). Here, it should be noted that any types of devices capable ofgenerating and outputting medical images may be used as the imagegeneration unit or the image generation device. For example, an X-rayimaging device such as an FPD (Flat Panel Detector) device, a CR(Computed Radiography) device or a CT (Computed Tomography) device, anMRI (Magnetic Resonance Imaging) device, a US (Ultra Sonic) echo device,or the like may be used. In any case, the image generation unit 1 neednot necessarily be limited to the above imaging (or radiography) device.That is, the image generation unit 1 may be the file server 300 of FIG.2 which stores the images generated by a not-shown imaging device.

Moreover, in FIG. 2, the hard disk 104 of the computer 100 or the fileserver 300 connected to the computer 100 through the network 400 can beused as the image storage unit 2. However, the present invention is notlimited to this. That is, a storage such as a hard disk built in theimage generation device 200 may be used as the image storage unit.

Image Processing Method

Subsequently, the whole operation in the image processing methodaccording to the present invention will be explained in detail withreference to the flow chart shown in FIG. 3. In the following, theexplanation premises that the CPU 101 shown in FIG. 2 controls the wholeprocessing operation and further executes the program module to achievethe relevant operation necessary for each constituent component.However, the present invention is not limited to this. That is, thepresent invention is applicable to various modes as described above.

Step S100

In response to an input by a user, the display control unit 6 readspatient information which is attached to the image of a certain specificpatient being the target of radiography. Here, it should be noted thatthe relevant image of the patient was generated by the image generationunit 1 and stored in the image storage unit 2. Then, the display controlunit 6 displays the read patient information on the image display unit7. In the present embodiment, as shown in FIGS. 4A and 4B, dates andhours of radiography for the certain specific patient, radiographyclassifications (regions, line-of-vision directions, etc.) and the likeare listed as the patient information. Here, the patient information maybe attached to the image data stored in the image storage unit 3 or maybe read from a database or the like administrated independently of theimage data. In FIGS. 4A and 4B, symbol C denotes a mouse cursor, andsymbols B1 and B2 denote buttons which are displayed so as to designateat least one pair of the images necessary for generating the temporaldifference image and can be activated when a user shifts the mousecursor C thereto and depresses (or clicks) it.

Step S110

The user first depresses the reference image button B1 to select themode of selecting an arbitrary reference image, and then selects theline corresponding to the image to be treated as the reference imagefrom the list displayed above the button B1. Here, in case of actuallyselecting the reference image, the user only has to shift the mousecursor C onto the intended line and then click it. FIG. 4A shows thestatus at that time. More specifically, FIG. 4A shows the status thatthe lowest line corresponding to the latest image is selected as thereference image from the displayed list. In this connection, the mark“S” indicating that the relevant line is selected as the reference imageis displayed on the left side of the selected line. At that time, it isset not to be able to simultaneously select two or more reference imagesfrom the displayed list. Thus, even if the user attempts to select thesecond line corresponding to the second reference image, the previouslyselected line is automatically set to be unselected.

Step S120

The user next depresses the comparison image button B2 to select themode of selecting an arbitrary comparison image, and then selects one ormore comparison images in the manner same as that for the referenceimage. At that time, the image designation unit 5 designates theselected image as the comparison image. In any case, the mark “R”indicating that the relevant line is selected as the comparison image isdisplayed on the left side of each of the selected lines. Here, althoughthe selectable reference image is only one, it is possible to select theplural comparison images. For example, in FIG. 4B, the three imagescorresponding to the second to fourth lines are selected as thecomparison images.

Step S130

The image comparison unit 3 compares the dates and hours of radiographywith respect to each pair of the selected reference and comparisonimages, determines order of difference in the later-described differenceprocess, and outputs the determined order to the difference processingunit 4. Incidentally, to acquire the dates and hours of radiography maybe based on the information attached to the selected image data or basedon the information of the database stored independently of the imagedata.

Here, the order of difference is determined as follows. That is, basedon the dates and hours of radiography of the target pair of thereference and comparison images, it is set to subtract the image of thelater date and hour of radiography from the image of the earlier dateand hour of radiography, or it is alternatively set to subtract theimage of the earlier date and hour of radiography from the image of thelater date and hour of radiography. Then, the set order is stored in theimage comparison unit 3.

In the present embodiment, it is assumed to set to subtract the image ofthe later date and hour of radiography from the image of the past(earlier) date and hour of radiography. However, the present inventionis not limited to this. That is, the reverse order is of courseacceptable.

As in the present embodiment, in a case where the date and hour ofradiography of the reference image is recent as compared with the datesand hours of radiography of the comparison images, the order ofdifference which is determined by the image comparison unit 3 is asshown in FIG. 6A. That is, if the reference image is given as S and thecomparison images are given as R1 to R3, the difference processing unit4 operates to generate the difference images TS1 to TS3 which satisfythe following equations.

TS1=R1−S   (1)

TS2=R2−S   (2)

TS3=R3−S   (3)

Step S140

The difference processing unit 4 reads one pair of the reference andcomparison images from the image storage unit 2, and generates andoutputs the difference image through the process as suggested in FIG.11. That is, a registration process is executed to deform the comparisonimage so as to conform the location of the comparison image to that ofthe reference image. Here, it should be noted that the reference imageis not deformed but only the comparison images are deformed. Then, thedifference processing unit 4 executes the difference process (differenceoperation) between the reference image and the deformed comparisonimages. Here, it should be noted that the reference image is subtractedfrom each of the deformed comparison images in the order of differenceoperation as shown in the above equations (1) to (3) and FIG. 6A. Inother words, the difference image is generated by subtracting the imageof the later date and hour of radiography from the image of the earlierdate and hour of radiography.

Step S150

The difference process is executed with respect to each pair of thereference and comparison images. Accordingly, when there are the threecomparison images as shown in FIGS. 4A and 4B, it is judged whether ornot the processed pair is the last pair every time each differenceprocess ends. If it is judged that the processed pair is the last pair,the flow advances to a next step S160. Meanwhile, if it is judged thatthe processed pair is not the last pair, the flow returns to theprevious step S130 to again execute the difference process.

Step S160

As described above, the generated three difference images are outputfrom the display control unit 6 and then displayed on the image displayunit 7, whereby the displayed difference images are used in diagnosis.

Modification 1

In the above, the case where the date and hour of radiography of thereference image is recent as compared with the dates and hours ofradiography of the comparison images is explained. On the contrary, acase where the date and hour of radiography of the reference image ispast as compared with the dates and hours of radiography of thecomparison images will be explained hereinafter. In the following, aswell as the first embodiment, the modification 1 will be explained withreference to the flow chart shown in FIG. 3.

Step S110

FIG. 5A shows the status that the reference image is selected. Morespecifically, FIG. 5A shows the status that the highest linecorresponding to the earliest image is selected as the reference imagefrom the displayed list. In this connection, the mark “S” indicatingthat the relevant line is selected as the reference image is displayedon the left side of the selected line.

Step S120

As well as the first embodiment, the user selects the comparison imageby using the mouse. At that time, the mark “R” indicating that therelevant line is selected as the comparison image is displayed on theleft side of each of the selected lines. For example, in FIG. 5B, thethree images corresponding to the third to fifth lines are selected asthe comparison images. Here, the comparison images are all selectedafter the reference image is selected.

Step S130

The image comparison unit 3 compares the dates and hours of radiographywith respect to each pair of the selected reference and comparisonimages, determines order of difference in the later-described differenceprocess, and outputs the determined order to the difference processingunit 4.

In the present embodiment, the comparison images are newly radiographedas compared with the reference image, the order of difference which isdetermined by the image comparison unit 3 is as shown in FIG. 6B. Thatis, if the reference image is given as S and the comparison images aregiven as R1 to R3, the difference processing unit 4 operates to generatethe difference images TS1 to TS3 which satisfy the following equations.

TS1=S−R1   (4)

TS2=S−R2   (5)

TS3=S−R3   (6)

Step S140

As well as the above explanation, the difference processing unit 4generates and outputs the difference image. The registration process isexecuted to deform the comparison image so as to conform the location ofthe comparison image to that of the reference image. Here, it should benoted that each of the deformed comparison images is subtracted from thereference image in the order of difference operation as shown in theabove equations (4) to (6) and FIG. 6B, thereby generating thedifference image.

Here, since the following steps are the same as those already explained,the explanation thereof will be omitted.

By executing the above process, it is possible to effectively generatethe difference images even if there are plural combinations of theimages to be compared. In addition, it is always possible to uniquelymaintain the relation between a signal in the difference image and achange actually occurred between the images.

FIGS. 7A and 7B are schematic diagrams which respectively show, asone-dimensional pixel value data, examples of the images and thetemporal differences in the above first embodiment and the modification1.

First, each of the upper four plots shown in FIG. 7A indicates the pixelvalue in the vicinity of the shadow of focus (seat of disease) in eachimage, and these four plots are corresponding to FIGS. 4A and 4B andFIG. 6A. More specifically, the shadow of focus becomes small graduallyfrom the comparison image R1 to the comparison image R3, and then theshadow of focus completely disappears in the reference image S which wasmost recently radiographed. Here, if the difference images TS1, TS2 andTS3 are generated as shown by FIG. 6A and the equations (1) to (3), thecondition of the change of the shadow is plotted as the differencesignal having the low pixel value as opposed to the background portionon which any change does not occur in the difference image. Themagnitude of the difference signal becomes small according as the shadowof focus becomes small. Then, in the difference between the comparisonimage R3 in which the shadow approximately disappears and the referenceimage S, the difference signal has the value substantially the same asthat of the background portion.

Meanwhile, with respect to the example shown in FIGS. 5A and 5B and FIG.6, the difference signal is acquired as shown in FIG. 7B. That is, inboth the cases, the shadow of focus becomes small gradually, and thedifference signal corresponding to the shadow of focus comes to have thelow value with respect to the background on which any change does notoccur. Incidentally, in case of displaying the above difference signal,for example, the signal value corresponding to the background portion isset to be equivalent to the median of the brightness level capable ofbeing displayed by a display device such as a CRT monitor or the like,and the display itself is executed by using, e.g., a gray region.

For example, in case of displaying the difference signal based on a grayscale, the difference signals respectively shown in FIGS. 7A and 7B aredisplayed as the black region as shown in FIG. 8A when the shadowdecreases. Meanwhile, the difference signals are displayed as the whiteregion as shown in FIG. 8B when the shadow increases. For this reason,even if the temporal relation is different between the standard andcomparison images, it is possible to maintain quite the same relation.Incidentally, it should be noted that the relation between theincrease/decrease of the shadow and the representation on the differenceimage is not limited to this. That is, for example, the gradationcharacteristics on the display may be inverted and displayed as shown inFIG. 8B.

Modification 2

In the above, the case where the date and hour of radiography of thereference image is recent as compared with the dates and hours ofradiography of the comparison images and the case where the date andhour of radiography of the reference image is past as compared with thedates and hours of radiography of the comparison images are explained.Besides, a case where the comparison images are radiographed before andafter the date and hour of radiography of the reference image will beexplained hereinafter as the modification 2 of the first embodiment. Inthe following, as well as the first embodiment, the modification 2 willbe explained with reference to the flow chart shown in FIG. 3.

Step S110

FIG. 13A shows the status that the reference image is selected. Morespecifically, FIG. 5A shows the status that the center line in thedisplayed list is selected as the reference image. In this connection,the mark “S” indicating that the relevant line is selected as thereference image is displayed on the left side of the selected line.

Step S120

As well as the above, the user selects the comparison image by using themouse. At that time, the mark “R” indicating that the relevant line isselected as the comparison image is displayed on the left side of eachof the selected lines. For example, in FIG. 13B, the four imagescorresponding to the first to second lines and the fourth to fifth linesare selected as the comparison images. Here, it should be noted that theselected comparison images include the comparison images selected bothbefore and after the reference image is selected.

Step S130

The image comparison unit 3 compares the dates and hours of radiographywith respect to each pair of the selected reference and comparisonimages, determines order of difference in the later-described differenceprocess, and outputs the determined order to the difference processingunit 4.

Here, the order of difference which is determined through thecombination of the first embodiment and the above modifications is asshown in FIG. 14A. That is, if the reference image is given as S and thecomparison images are given as R1 to R4, the difference processing unit4 operates to generate the difference images TS1 to TS4 which satisfythe following equations.

TS1=R1−S   (7)

TS2=R2−S   (8)

TS3=S−R3   (9)

TS4=S−R4   (10)

In the above processing method, any problem does not occur seemingly inthe case where the shadow of focus uniformly decreases or uniformlyincreases. However, if chemical therapy is executed in the actualmedical treatment, the shadow of focus tends to repeatedly increase anddecrease. In such a case, regardless of the temporal relation betweenthe reference image and the comparison image, to subtract the past imagefrom the reference image is easy to be understood in the diagnosis. Thatis, when the date and hour of radiography of the reference image isearlier than the date and hour of radiography of the comparison image,the subtraction in the difference process is executed in the directionsame as that in the above modification. On the contrary, when the dateand hour of radiography of the reference image is later than the dateand hour of radiography of the comparison image, the difference processis executed in the direction opposite to that in the first embodiment.In the modification 2, the order of difference which is determined bythe image comparison unit 3 is as shown in FIG. 14B. That is, if thereference image is given as S and the comparison images are given as R1to R4, the difference processing unit 4 operates to generate thedifference images TS1 to TS4 which satisfy the following equations.

TS1=S−R1   (11)

TS2=S−R2   (12)

TS3=S−R3   (13)

TS4=S−R4   (14)

Step S140

As well as the above explanation, the difference processing unit 4generates and outputs the difference image. The registration process isexecuted not to deform the reference image but to deform the comparisonimage so as to conform the location of the comparison image to that ofthe reference image. Here, it should be noted that each of the deformedcomparison images is subtracted from the reference image in the order ofdifference operation as shown in the above equations (11) to (14) andFIG. 14B, thereby generating the difference image.

Here, since the following steps are the same as those already explained,the explanation thereof will be omitted.

By executing the above process, it is possible to effectively generatethe difference images even if there are plural combinations of theimages to be compared. In addition, it is always possible to uniquelymaintain the relation between a signal in the difference image and achange actually occurred between the images.

FIGS. 15A and 15B are schematic diagrams which respectively show, asone-dimensional pixel value data, examples of the images and thetemporal differences in the modification 2.

First, each of the upper five plots shown in FIG. 15A indicates thepixel value in the vicinity of the shadow of focus in each image, andthese five plots are corresponding to FIGS. 13A and 13B and FIG. 14A.More specifically, the shadow of focus becomes small gradually from thecomparison image R1 to the comparison image R4, and then the shadow offocus completely disappears in the reference image S which was mostrecently radiographed. Here, if the difference images TS1, TS2 and TS4are generated as shown by FIG. 14A and the equations (7) to (10), themagnitude of the difference signal becomes small according as it comesclose to the magnitude of the shadow of focus in the reference image.However, the condition of the change of the shadow is plotted as thedifference signal having the wholly low pixel value as opposed to thebackground portion on which any change does not occur in the differenceimage.

Meanwhile, with respect to the example shown in FIGS. 13A and 13B, thedifference signal is acquired as shown in FIG. 15B. That is, in thecomparison image of which the shadow of focus is smaller than that ofthe reference image, the difference signal corresponding to the shadowof focus has the low value with respect to the background on which anychange does not occur. Meanwhile, in the comparison image of which theshadow of focus is larger than that of the reference image, thedifference signal corresponding to the shadow of focus has the highvalue with respect to the background on which any change does not occur.Incidentally, in case of displaying the above difference signal, forexample, the signal value corresponding to the background portion is setto be equivalent to the median of the brightness level capable of beingdisplayed by a display device such as a CRT monitor or the like, and thedisplay itself is executed by using, e.g., a gray region.

For this reason, the difference images generated by the equations (4) to(14) in the modifications 1 and 2 can be displayed by only executing ablack/white reversal process to the difference images generated by theequations (1) to (3). In this connection, it is also possible topreviously generate the difference images all in the same direction, andthen execute the black/white reversal process according to the temporalrelation of the reference and comparison images.

Moreover, in the different process of the present invention, thecomparison image is always deformed for the registration irrespective ofthe temporal relation of the reference and comparison images. Therefore,in case of interpreting the difference image generated from a series ofthe comparison images, it is possible to always generate the differenceimage based on the orientation (posture) of the subject in one referenceimage, whereby it is possible to easily interpret the generateddifference image.

FIGS. 12A and 12B are diagrams showing examples of the above situation.Here, it should be noted that FIG. 12A is corresponding to FIGS. 4A and4B, and FIG. 12B is corresponding to FIGS. 5A and 5B.

That is, FIGS. 12A and 12B show the basic of the above difference imageprocess in more detail. More specifically, the deformed images R1w toR3w all registered with respect to the reference image S are generatedin relation to the comparison images R1 to R3. These deformed images aretemporarily generated inside the difference processing unit 4 anddirectly used in the difference process.

In FIGS. 12A and 12B, it should be noted that each deformed image hasbeen registered with respect to the reference image S. For this reason,even if the orientation of the subject in each radiographed imagediffers from others, the orientation of the subject in each of thedeformed images R1w to R3w conforms to the reference image S.

In other words, in FIG. 12A, since each of the comparison images R1 toR3 has been resultingly registered with respect to the reference imageS, the orientation of the subject in the difference image is the same asthat in the reference image S. Meanwhile, as shown in FIG. 12B, even inthe case where the reference image S is radiographed in the earliest(i.e., May 20), the orientation of the subject in each difference imageis the same as that in the reference image S. For these reasons, evenwhere the plural difference images are compared with others, it ispossible to easily grasp the correspondence between the attendedportions, and it is thus possible to more easily execute the diagnosis.

Second Embodiment

In the first embodiment, the reference image and the comparison imagesare listed on the screen, and a user selects the arbitrary images fromthe displayed list. Otherwise, it is possible to display the imagesthemselves on the screen to enable the user to select the arbitraryimages as directly watching them.

FIG. 9 is a schematic diagram showing an example that a reduction imageis selected by using a mouse to select the reference image or thecomparison image.

In FIG. 9, the information indicating the test date and time or the likeis displayed next to the displayed relevant reduction image, and, whenthe reduction image is clicked, the clicked reduction image is set asthe reference image.

In any case, the method of arranging the reduction images is not limitedto this, that is, various methods are applicable. For example, asdisclosed in Japanese Patent Application Laid-Open No. 10-155746described above, it is possible to vertically and horizontally arrange aseries of time-series images in a selectable manner.

Moreover, in case of selecting the reference image and the comparisonimages, it is possible not to provide any selection button such as theabove buttons B1 and B2. In that case, it is possible to select thetarget image by using plural buttons attached to the mouse. In addition,in case of selecting the plural comparison images, it is possible toappropriately combine mouse and keyboard operations. In that case, forexample, the user clicks the mouse as depressing a specific key toselect the plural images.

Third Embodiment

In the first and second embodiments, the screen used for selecting theimages is independently provided, and the reference image and thecomparison images are selected by using this screen. However, thepresent invention is not limited to this. That is, it is possible toselect the target images on the screen typically used for executingimage diagnosis.

FIGS. 10A and 10B are schematic diagrams showing an example of imagedisplay on the image display unit 7 in the diagnosis.

In FIG. 10A, there are four image display regions in a window W, and theimages of an identical patient are displayed sequentially from the upperleft region to the lower right region in order of recent (latest)radiography time. Here, if the region which is clicked and of which theframe is thus emphatically displayed is set as the reference image andother regions are set as the comparison images, it is possible toappropriately select the images according to their display locations onthe screen.

That is, when the latest (current) image is set as the reference imageand the remaining three images are set as the comparison images as shownin FIG. 6A, it only has to select the upper left region as the referenceimage region as shown in FIG. 10A. Meanwhile, when the earliest (mostpast) image is set as the reference image as shown in FIG. 6B, if thedisplay region of the most-past image is selected from the display imageregions, it is possible to resultingly select the image of the selectedregion as the reference image.

As explained above, according to the present embodiment, in case ofgenerating the temporal difference image based on the pluralcombinations of the images, even if the method of combining the imagesto be compared and interpreted is changed, it is possible toconsistently extract the change of the shadow of focus. In addition, itis possible to effectively generate the difference image with theorientation of the subject maintained. As a result, it is possible toachieve the image process which can execute the effective differenceprocess in progress observation.

Other Embodiments To Which the Present Invention Is Applied

It should be noted that the image comparison unit 3, the differenceprocessing unit 4, the image designation unit 5 and the display controlunit 6 which are the main constituents of the image processing deviceaccording to the above embodiments, and the respective steps (e.g.,steps S110 to S160 shown in FIG. 3) of the image processing methodsaccording to the above embodiments can be achieved by operating theprograms stored in the RAM, the ROM or the like of a computer. In thisconnection, the relevant programs and a computer-readable storage mediumwhich stores therein the relevant programs are included in the conceptof the present invention.

Moreover, the relevant programs are supplied to the computer through thestorage medium such as, e.g., a CD-ROM or through various transmissionmedia. As the storage medium for recording the relevant programs, forexample, a flexible disk, a hard disk, a magnetic tape, a magnetoopticaldisk, a nonvolatile memory card or the like can be used as well as theCR-ROM. In addition, as the transmission media for supplying therelevant programs, communication media (a wired line such as an opticalfiber, a wireless line, etc.) in a computer network (a LAN (local areanetwork), a WAN (wide area network) such as the Internet, a wirelesscommunication network, etc.) system for transmitting program informationas carrier waves to supply the relevant programs can be used.

Moreover, the present invention includes not only a case where thefunctions of the above embodiments are achieved when the computerexecutes the supplied programs, but also a case where an OS (operatingsystem) or the OS in association with other application softwarefunctioning on the computer achieves the functions of the aboveembodiments. In addition, the present invention also includes a casewhere a function expansion board inserted in the computer or a functionexpansion unit connected to the computer executes a part or all of theactual processes based on the supplied programs, the functions of theabove embodiments are thus achieved. In other words, in such cases, therelevant programs are included in the present invention.

In the main constitution of the image processing device, each unit canbe implemented as the logical constituent element in one program or anindependent library. The relevant program is stored in such a hard diskHD as shown in FIG. 2. Then, the storage program is read in response toa not-shown user input or another indication input, the read program isuncompressed on the RAM, and the program is sequentially executed by theCPU, thereby achieving the functions of the above embodiments.Alternatively, the program may be stored in the ROM or a file server FSconnected to the device through a network N.

Moreover, a part or all of the above constituent element can be achievedby such hardware as an accelerator ACC shown in FIG. 2. Morespecifically, it is possible to connect the accelerator ACC to the busBUS as the hardware including the difference processing unit 4 and otherfunctions, whereby it is also possible to cause the CPU to execute thewhole control of the relevant hardware.

1-20. (canceled)
 21. An image processing device comprising: a storageunit adapted to store plural images; an image designation unit adaptedto designate a reference image and comparison images from among theplural images stored by said storage unit; a comparison unit adapted toacquire radiography date and time information of the designatedreference image and the designated comparison images, and compare theradiography date and time information of the reference image with theradiography date and time information of each of the comparison images;a determination unit adapted to determine an operation in case ofgenerating a temporal difference image based on each of the comparisonresults by said comparison unit; and a difference image generation unitadapted to generate the temporal difference image from the referenceimage and each of the comparison images by using the operation,corresponding to the each of the comparison images, determined by saiddetermination unit.
 22. An image processing device according to claim21, wherein said determination unit determines, based on each of thecomparison results by said comparison unit, whether to subtract thecomparison image from the reference image or subtract the referenceimage from the comparison image.
 23. An image processing deviceaccording to claim 21, wherein said difference image generation unitdeforms each of the comparison images on the basis of the referenceimage designated by said image designation unit, and generates thetemporal difference image from the reference image and the deformedcomparison image.
 24. An image processing device according to claim 21,wherein said image designation unit can display a list of the imagesstored by said image storage unit, and selects the reference image andthe comparison images from the displayed list of the images.
 25. Animage processing device according to claim 21, wherein said imagedesignation unit can display reduction images acquired by reducing theimages stored by said storage unit, and selects the reference image andthe comparison images based on an indication input with respect to thedisplayed reduction images.
 26. An image processing device according toclaim 21, wherein said image designation unit can display the imagesstored by said storage unit in predetermined order based on theradiography date and time information, and selects the reference imageand the comparison images based on an indication input with respect tothe displayed images.
 27. An image processing device according to claim21, wherein said difference image generation unit determines, based onthe comparison result by said comparison unit, the operation so as tosubtract the image of which the radiography date and time informationindicates more recent date and time from the image of which theradiography date and time information indicates more past date and time.28. An image processing method comprising: an image designation step ofdesignating a reference image and comparison images from among pluralimages stored in a storage unit; a comparison step of acquiringradiography date and time information of the designated reference imageand the designated comparison image, and comparing the radiography dateand time information of the reference image with the radiography dateand time information of each of the comparison images; a determinationstep of determining an operation in case of generating a temporaldifference image based on each of the comparison results in saidcomparison step; and a difference image generation step of generatingthe temporal difference image from the reference image and each of thecomparison images by using the operation, corresponding to the each ofthe comparison images, determined in said determination step.
 29. Animage processing method according to claim 28, wherein saiddetermination step is adapted to determine, based on each of thecomparison results in said comparison step, whether to subtract thecomparison image from the reference image or subtract the referenceimage from the comparison image.
 30. An image processing methodaccording to claim 28, wherein said difference image generation step isadapted to deform each of the comparison images on the basis of thereference image designated in said image designation step, and generatethe temporal difference image from the reference image and the deformedcomparison image.
 31. An image processing method according to claim 28,wherein said image designation step is adapted to display a list of thestored images, and select the reference image and the comparison imagesfrom the displayed list of the images.
 32. An image processing methodaccording to claim 28, wherein said image designation step enables thedisplay of reduction images acquired by reducing the stored images, andis adapted to select the reference image and the comparison images basedon an indication input with respect to the displayed reduction images.33. An image processing method according to claim 28, wherein said imagedesignation step enables the display of the stored images in apredetermined order based on the radiography date and time information,and is adapted to select the reference image and the comparison imagesbased on an indication input with respect to the displayed images. 34.An image processing method according to claim 28, wherein saiddifference image generation step is adapted to determine, based on thecomparison result in said comparison step, the operation so as tosubtract the image of which the radiography date and time informationindicates more recent date and time from the image of which theradiography date and time information indicates more past date and time.35. A computer-readable storage medium which stores a control programincluding: an image designation step of designating a reference imageand comparison images from among plural images stored in a storage unit;a comparison step of acquiring radiography date and time information ofthe designated reference image and the designated comparison image, andcomparing the radiography date and time information of the referenceimage with the radiography date and time information of each of thecomparison images; a determination step of determining an operation incase of generating a temporal difference image based on each of thecomparison results in said comparison step; and a difference imagegeneration step of generating the temporal difference image from thereference image and each of the comparison images by using theoperation, corresponding to the each of the comparison images,determined in said determination step.
 36. A computer-readable storagemedium according to claim 35, wherein said determination step is adaptedto determine, based on each of the comparison results in said comparisonstep, whether to subtract the comparison image from the reference imageor subtract the reference image from the comparison image.
 37. Acomputer-readable storage medium according to claim 35, wherein saiddifference image generation step is adapted to deform each of thecomparison images on the basis of the reference image designated in saidimage designation step, and generate the temporal difference image fromthe reference image and the deformed comparison image.
 38. Acomputer-readable storage medium according to claim 35, wherein saidimage designation step is adapted to display a list of the storedimages, and select the reference image and the comparison images fromthe displayed list of the images.
 39. A computer-readable storage mediumaccording to claim 35, wherein said image, designation step enables thedisplay of the stored images in a predetermined order based on theradiography date and time information, and is adapted to select thereference image and the comparison images based on an indication inputwith respect to the displayed images.
 40. A computer-readable storagemedium according to claim 35, wherein said difference image generationstep is adapted to determine, based on the comparison result in saidcomparison step, the operation so as to subtract the image of which theradiography date and time information indicates more recent date andtime from the image of which the radiography date and time informationindicates more past date and time.